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ABSTRACT 

In 1999 the Texas Legislature provided for the creation of 
the Master Reading Teacher (MRT) Certificate to address several issues. MRTs 
primary duties are to: serve as reading teachers; consult with and mentor 
other teachers; and provide research-based information and training. In 
September 2001 the Texas Education Agency asked Dr. Jan Hasbrouck to conduct 
an evaluation of MRT training that occurred during the 2000-2001 academic 
year. There were five evaluation questions, but this paper discusses only two 
questions: To what extent have changes in service delivery due to MRT 
training had a positive effect on student standardized reading test scores 
(both for students directly taught by MRT program graduates and for students 
indirectly influenced by MRT program graduates) ; and From perceptions of both 
administrators and MRTs, what are the MRT program’s greatest strengths and 
weaknesses? The paper states that the evaluation consisted of four 
activities: (1) statewide mail survey of all MRTs currently receiving the 

state stipend for MRTs serving on *'high-need” campuses; (2) mail survey of 
the administrators of MRTs drawing a state stipend; (3) telephone follow-up 
survey of selected MRTs and administrators; and (4) Texas Assessment of 
Academic Skills (TAAS, the criterion referenced tests required of Texas 
public school students from the third-tenth grade) data collection by 
selected administrators. According to the paper, evaluators gained the 
impression of a moderately strong beginning of the MRT program, but a 
beginning that fell short of its potential. With better networking and 
improved guidelines on program implementation, the prediction is for stronger 
programs. (NKA) 
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Background 

In 1999, the Texas Legislature provided for the creation of the Master Reading Teacher 
(MRT) Certificate to address several issues. The MRT certification program was developed as 
part of the Texas Reading Initiative to help all students 

• Read on level by the end of the 3rd grade 

• Increase their reading knowledge and skills throughout their school careers 

• Receive instruction from highly qualified teachers 
The MRT’S primary duties are to: 

• Serve as a reading teacher 

• Consult with and mentor other teachers 

• Provide research-based information and training 

Review of the Literature 

A growing body of research indicates that many reading difficulties are preventable if 
students receive support and intervention, especially early intervention (Allington & 
Cunningham, 2002; Snow, Bums, & Griffin, 1998; Torgesen et al., 1999). Developing and 
supporting teacher expertise is a key element in reading success. Training and support for 
teachers directly involved in delivery of instmction is vital to develop, support, and maintain this 
level of teaching expertise. The US. Department of Education has listed several key similarities 
among successful reading programs, including effective staff development for teachers and a 
significant amount of time spent on basic instmction (Turnbull, Welsh, Held, Davis, & 
Ratnofsky, 1999). 

In a time of increased school accountability, teacher recmitment and retention issues, and 
increased diversity in public schools, the Texas MRT model is specifically designed to support 
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student learning and teacher needs. Early evaluation of the implementation of this model is 
beneficial to other educators, state education agencies, and educational policymakers as they 
consider similar programs. This paper presents the results of a State-initiated evaluation of the 
impact of MRTs on student performance. 

Evaluation Questions 

In September 2001 the Texas Education Agency (TEA) asked Jan E. Hasbrouck, Ph.D., 
Associate Professor, Department of Educational Psychology, Texas A&M University (TAMU), 
to conduct an evaluation of MRT training that occurred during the 2000-2001 academic year. 
The evaluation was to answer four questions, and the evaluation team added a fifth. However, 
this paper discusses only the following two questions: 

1. To what extent have changes in service delivery due to MRT training had a 
positive effect on student standardized reading test scores (both for students 
directly taught by the MRT program graduate and for students indirectly 
influenced by the MRT program graduate). 

2. From perceptions of both administrators and MRTs, what are the MRT 
program’s greatest strengths and weaknesses? 

Method 

The evaluation consisted of four separate activities: 

1. Statewide mail survey of all MRTs who are currently receiving the State 
stipend for MRTs serving on “high-need” campuses. 

2. Mail survey of the administrators of MRTs drawing a State stipend. 

3. Telephone follow-up survey of selected MRTs and administrators. 
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4. Texas Assessment of Academic Skills (TAAS, the Texas criterion referenced 
tests required of Texas public school students from the 3"* to the 10* grade) data 
collection by selected administrators. 

Mail Survey 

Texas offered State stipends of $5,000 annually for three years (fall, 2000 - spring, 2003) 
to teachers who serve as MRTs on one of the more than 1800 campuses identified by the TEA as 
“high-need” campuses. By January 2002 the Texas State Board for Educator Certification had 
certified 828 MRTs statewide; 263 of these MRTs were eligible for State “high-need” campus 
stipends. District, campus, and MRTs’ names for only these 263 “high-needs” MRTs were 
available, so these individuals and their administrators constituted the sample for the evaluation. 
Follow-up Telephone Survey 

In the teacher mail survey teachers were asked if they would participate in a follow-up 
telephone survey. Twenty-five MRTs were selected at random from among those who 
volunteered to participate in a follow-up survey. In addition, 25 administrators were selected at 
random from among administrator respondents to participate in the follow-up. 

TAAS Data Collection by Selected Administrators 

The administrator mail survey asked administrators if they would provide certdn TAAS 
data from their campus. Those who indicated they would provide these data were contacted and 
given additional information and data reporting sheets. 

Procedure 

Mail surveys. In April 2002 the team mailed surveys to 263 teachers and 245 
administrators. A total of 198 teachers (75.3% response rate) and 159 administrators (64.9% 
response rate) responded to the mail questionnaires. 
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Follow-up telephone survey. Responses to mail surveys were analyzed, and a short 
questionnaire was formulated to further probe topics that seemed to generate special interest 
among both teachers and administrators. Over a period of several weeks 18 administrators and 
24 teachers from 40 campuses were administered the questionnaire by way of an individual 
telephone interview. 

TAAS data collection by selected administrators. Administrators who indicated in their 
mail survey that they were willing to provide TAAS data received a form that asked for the 
individual student and class TAAS performance data that is provided in the TEA reports to the 
schools. Ten administrators responded with TAAS data. 

Results Summary 

Evaluation question 1. To what extent have changes in service delivery due to MRT 
training had a positive effect on student reading scores (both for students directly taught by the 
MRT graduate and for students indirectly influenced by the MRT graduate). 

Survey Data 

A fair evaluation of the MRT program by student reading scores at this time was 
considered to be problematic by a majority (63% of item respondents) of administrators. On 
whether TAAS scores could now be used to fairly evaluate the MRT program, administrators 
were evenly split, 50% “Yes” and 50% “No.” 

Ninety-nine administrators provided 133 reasons why the MRT program could not yet be 
fairly evaluated: 40% noted that the program was too new to evaluate, 29% noted that the MRT 
program had been only partially implemented, and 23% felt they had insufficient student data for 
a fair evaluation. Finally, 20% noted that the MRT program began at the same time as other 
innovations, so its effects could not be fairly isolated. 
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Despite this hesitancy for an early MRT program evaluation based on student scores, 
both administrators and MRTs expressed a widespread perception that the MRT on campus had a 
positive effect on student Texas Primary Reading Inventory (TPRI) and TAAS scores. When 
administrators were asked specifically whether TPRI or TAAS scores had increased due to MRT 
direct teaching, 71% of item respondents said, “Yes,” and only 29% said “No.” When asked by 
how much these scores improved, 57% of item respondents reported “up to 10 points 
improvement in scores (on TPRI and/or TAAS),” an additional 19% reported up to 20 points 
improvement in scores,” and an additional 13% reported great gains of “up to 30 points 
improvement in scores.” 

Administrators were also asked about TPRI or TAAS score increases due to indirect 
MRT support (enhancing instruction by others). Responses to this question closely mirrored 
those for MRT direct teaching. 

TAAS Data from Administrators 

Administrators were asked in the survey if they would provide TAAS data from their 
school from 2001 and 2002. A number of administrators replied in the affirmative, and 
administrators provided data on the TAAS reading performance of 139 students in 2001 (before 
they were instructed by an MRT) and in 2002 (after they had been instructed by an MRT), e.g., 
grade 3 in 2001 and grade 4 in 2002. 

Individual student data (not aggregated). Student skill growth may be viewed at the 
individual student level or at the aggregate school level. First are presented individual student 
growth data. (Graph 1) TAAS reading scores averaged for all 139 students increased 5.5 
percentage points from 73.9 Texas Learning Index (TLI) score to 79.4 TLI. The range of the 
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gains was 83, from -30 to +53. There were 102 increased scores and 32 decreased scores; five 
scores stayed the same from one exam administration to the next. 




TA>^8goie knpiunnlBy IndluSklnlSanpled 

This box plot shows the increase in the median score, 25th, and 75th percentile scores, 
from 2001-2002. 

Of the 139 students in the reported sample, the number meeting minimum TAAS 
standards increased by 23, from 93 in 2001 to 1 16 in 2002. The number of students mastering all 
objectives rose by 13, from 28 in 2001 to 41 in 2002. 

Performance aggregated by school. (Graph 2). The performance changes in the 
individual TAAS score sample averaged by school ranged from -5.5 percentage points to +14 
percentage points. (NOTE: The number of students reported per school ranged from 2 to 51 with 
an average of 14.) So, although the presence of an MRT may have had a positive effect on TAAS 
reading scores in most cases, it was not true for all schools. 
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The box plot shows that average school aggregate median score and 75th %ile ranks 
improved by approximately 10 TLI score points. However, 25th %ile ranks stayed about the 
same, and 10th percentile ranks dropped in TAAS performance. In other words, the growth seen 
for middle and high performing students was not evident for the lowest 25% of students. This 
uneven growth for high vs. medium and low performers caused a spreading out of the score 
distribution in year 2002. 

Nine administrators provided data on 1 5 pairs of classes that showed the percentage of 
students meeting minimum expectations on the 2001 TAAS reading in one grade and the 
percentage of students meeting minimum expectations on the 2002 TAAS reading in the next 
higher grade, e g., grade 3 in 2001 and grade 4 in 2002. (Graph 3) The average increase in 
passing rates was 10.9%, from 79.8% to 90.7%. Statewide the percentage of students passing the 
TAAS reading increased by 1.4%, from 88.9% to 91.3%. Therefore, the students in this sample 
of classes improved by a greater percentage than the State average. The range of sample class 
score changes was 32, from -4 (one case of a lower percentage of students passing) to +28 
percent. The box plot below shows the increased median level of percentage of students meeting 
minimum expectations in TAAS reading, as well as the reduced range of scores from 2001-2002. 
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Met lufn TAAS Rdg Rqmt 2DD1-2DD2 

The reader is cautioned that this grade-to-grade comparison does not imply that the data 
represent exactly the same cohorts in both 2001 and 2002. The groups of students going from 
one grade in 2001 to another grade in 2002 undoubtedly changed. However, many of the same 
students would continue in the same school from one year to the next and would be represented 
in the data for both 2001 and 2002. 

Supplemental Evaluation Question 5: According to both administrators and MRTs, what 
are the MRT program’s greatest strengths and weaknesses? 

Administrators and MRTs were asked similar questions, both in open-ended wnte-in 
format. Administrators were asked to identify two or three aspects of their MRT program which 
were strengths and two or three areas of needed improvement. MRTs were asked for two or three 
things about the program they liked most, and two or three suggestions for improvement. 

MRTs volunteered 12 different strengths that were shared by multiple respondents (and 
22 others unique to individual respondents). The top four strengths were each listed by at least 
38 respondents. They are (in order of descending popularity): a) Professional development, b) 
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Opportunity to collaborate with others, c) Positively impacting student learning, and d) 
Mentoring/Consulting with peers. 

MRTs volunteered 10 different improvement needs shared by multiple respondents (and 
25 others unique to individual respondents). The top four improvement needs were each listed by 
at least 39 respondents. They are (in order of descending popularity): a) Improved time 
scheduling, b) Ongoing training, c) Networking, and d) Better definition of the MRT role. 

Administrators volunteered a total of 10 different strengths of the MRT program, of 
which all but one were listed by multiple respondents. The most popular four strengths were each 
listed by at least 33 respondents. They are (in descending order of popularity): a) Expertise in 
mentoring/consulting, b) Having an additional campus resource, c) Support and training for new 
teachers, d) Having an additional support for students. 

Administrators also suggested a total of 29 different improvement needs for the MRT 
program, of which only 10 were listed by multiple respondents. Of these 10 suggestions, the 5 
most popular were each listed by at least 14 respondents. These 5 improvement ideas are (in 
descending order of popularity): a) Continue and expand funding (stipends, release time, 
curriculum materials, staffing allocations, etc.), b) Structure more flexible scheduling and release 
time, c) Clarify the MRT role, d) Train administrators as well as MRTs, e) Continued training of 
the MRT. 

Conclusion 

This evaluation of the MRT program relied on data that were descriptive in nature, and 
the design was correlational, which disallows causal inferencing fi’om TAAS data. Added to 
these limitations was the relatively small sample of schools and students for whom TAAS data 
were reported by administrators. Further reducing our ability to make causal inferences between 
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the MRT program and student reading gains was the fact that other reading improvement efforts 
began in several schools at the same time as the MRT program. It is therefore not possible to 
state that MRTs caused improvements in TAAS reading performance in the reporting schools. 
However, the combination of the TAAS scores reported here, the survey responses by 
administrators and teachers, and the follow-up phone interviews caused these evaluators to 
believe that MRTs were having positive effects. 

There were common program start-up problems, including perceived lack of direction 
and guidelines, and lack of time and flexibility within the school day for the MRTs to carry out 
the various roles for which they were trained. However, an overall image of four types of 
positive MRT effects pervaded this evaluation study. 

The first positive effect was upon the MRTs themselves. The training and some new role- 
related freedom were professionally invigorating for several. Though too challenging for a few, 
and somewhat fiiistrating for several, the overall impression was one of professional growth 
toward a new challenge and a desire for more training. 

The second positive effect was upon colleagues in the MRTs’ schools. Because other 
teachers were not directly interviewed or surveyed, this positive effect must be conjectured in 
part. However, perceptions by most administrators tended to agree with those of the MRTs — ^that 
positive effects on colleagues were common. In-servicing, professional advice/consultation, and 
mentoring of new teachers commonly occurred and was appreciated by most administrators. 

The third positive effect was upon the reading program (broadly defined) within the 
schools. Administrators noted a new energy or focus to reading, more training activity, and a 
sense of higher priority for reading. They also noted that this focus cut across the 
dyslexia/special/general education divides. Though perception of this positive effect was 
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widespread, it is difficult to attribute it to the MRT role alone. We can only conclude that the 
MRT role was perceived as an important element in a new or stronger or clearer focus on reading 
instruction. 

The fourth and final positive effect was upon student TAAS scores. As with the previous 
effect, this fourth effect cannot be readily attributed to the MRT program alone. Other, 
concurrent improvements commonly accompamed the MRT role, as one might hope for when 
undertaking thoughtfiil, integrative, school-wide change. In addition, it needs to be noted that 
student improvement was not universal — most, but not all, schools improved in student TAAS 
scores. 

In a most general sense, the evaluators gained the impression of a moderately strong 
beginning for the MRT program, but a beginning that fell short of meeting its potential. 
Individual MRTs felt themselves relatively isolated from their peers and without clear direction 
for their role. Administrators felt a lack of direction as well. Given better opportumty for 
networking, and for learning from model programs, and improved guidelines on program 
implementation, and another year of “settling in”, we predict stronger programs from which 
evaluators could reliably document student reading score improvement within an appropriate 
comparison design. 
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